Data extraction from form images

ABSTRACT

An image processing system accesses an image of a completed form document. The image of the form document includes one or more features, such as form text, at particular locations within the image. The image processing system accesses a template of the form document and computes a rotation and zoom of the image of the form document relative to the template of the form document based on the locations of the features within the image of the form document relative to the locations of the corresponding features within the template of the form document. The image processing system performs a rotation operation and a zoom operation on the image of the form document, and extracts data entered into fields of the modified image of the form document. The extracted data can be then accessed or stored for subsequent use.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/283,762, filed Feb. 23, 2019, now patent Ser. No. ______, which isincorporated by reference in its entirety.

BACKGROUND

This description generally relates to image processing and in particularto data extraction using image processing techniques.

In current systems, when a user uploads an image of a form document, ahuman operator must manually extract the data from the form document andenter it into an electronic database. In addition to manual dataextraction, human operators must also transform the image in order toproperly view the form. For example, when users upload images of formdocuments that are rotated and/or shifted within the image, too small toread, partly cut-off, a human operator must transform the image beforedata extraction and entry. As a result, manual image transformation anddata extraction are inefficient and are prone to inaccuracies.

SUMMARY

The quality of images of form documents can vary, for instance inorientation, size, image quality, and the like. An image processingengine accounts for and corrects this variation in order toautomatically extract data from form documents. To do this, the imageprocessing engine may transform the image of the form document using thelocation of pre-determined form document features within the image ofthe form documents and calibration parameters (e.g., a calibrationdistance and a calibration angle) determined based on a templatedocument. For instance, the image may be zoomed, rotated, and/or shiftedwhen the image is transformed. The image processing engine extracts someor all of the data from the transformed image of the form document. Theimage processing engine may also transform the coordinates of textwithin an image of a form document without transforming the image of theform document. The image processing engine extracts some or all of thedata from the image of the form document using the transformedcoordinates, for instance without necessarily performing thetransformation operations described above.

The image processing engine accesses an image of a form document. Imagesof form documents are provided by users of the image processing engine.Completed form documents are forms containing fields with form textassociated with the fields and text entered within the fields. Form textis text describing the field (e.g., “first name” and “address”), andentered text is text entered within the field by an entity completingthe form (e.g., the text “Nathan” within a “name” field and the text“123 A Street” within an “address” field). A first distance between afirst set of form text and a second set of form text is determined oraccessed. Similarly, a first angle between the first set of form textand the second set of form text is determined or accessed. The firstdistance and the first angle are used as calibration parameters in orderto transform the image of the form document, and they may be determinedusing a template form document.

Image coordinates are determined for each of the first set of form textand second set of form text. Image coordinates may be determined by theimage processing engine or by third-party software. The imagecoordinates may represent a perimeter of a bounding box around formtext, an edge of a bounding box around the form text, a corner of abounding box around the form text, etc. The image processing enginecomputes a second distance between the set of image coordinatescorresponding to the first set of form text and the set of imagecoordinates corresponding to the second set of form text. Similarly, asecond angle of a line intersecting the set of image coordinates betweenthe set of image coordinates corresponding to the first set of form textand the set of image coordinates corresponding to the second set of formtext is computed by the image processing engine. In some embodiments,transformation operations, including a zoom operation and a rotationoperation, are performed on the image of the form document based on acomparison between the first distance and second distance and based on acomparison between the first angle and second angle to produce amodified form document image. In other embodiments, transformationoperations are performed on the image coordinates of the form text andentered text of the image of the form document. Form text and enteredtext may then be extracted from the image of the form document.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a diagram of a system environment of an imageprocessing engine, according to one embodiment.

FIG. 2 is a block diagram of an architecture of the image processingengine, according to one embodiment.

FIGS. 3A-3D illustrate the process of modifying an image of a formdocument, according to one embodiment.

FIG. 4 is a flow chart illustrating a method of extracting data from animage of a form document, according to one embodiment.

The figures depict various example embodiments of the present technologyfor purposes of illustration only. One skilled in the art will readilyrecognize from the following description that other alternativeembodiments of the structures and methods illustrated herein may beemployed without departing from the principles of the technologydescribed herein.

DETAILED DESCRIPTION System Overview

FIG. 1 illustrates a diagram of a system environment 100 of an imageprocessing engine 125, according to one embodiment. The systemenvironment 100 shown in FIG. 1 includes a user 105, a client device110, a scanning engine 115, a network 120, and an image processingengine 125. In alternative configurations, different and/or additionalcomponents may be included in the system environment 100.

A user 105 of the image processing engine 125 may be an individual ororganization. A user 105 may use the image processing engine 125 toextract text from form documents. For example, a user 105 may havehardcopies of form documents for employees, volunteers, customers and/orpersonnel and may want to translate text from the hardcopies into anelectronic database.

The client devices 110 are one or more computing devices capable ofreceiving user input as well as transmitting and/or receiving data via anetwork 120. In one embodiment, a client device 110 is a conventionalcomputer system, such as a desktop or a laptop computer. Alternatively,a client device 110 may be a device having computer functionality, suchas a personal digital assistant (PDA), a mobile telephone, a smartphone,or another suitable device. A client device 110 is configured tocommunicate via the network 120. In one embodiment, a client device 110executes an application allowing a user of the client device 110 tointeract with the image processing engine 125. For example, a clientdevice 110 executes a browser application to enable interaction betweenthe client device 110 and the image processing engine 125 via thenetwork 120. In another embodiment, a client device 110 interacts withthe image processing engine 125 through an application programminginterface (API) running on a native operating system of the clientdevice 110, such as IOS® or ANDROID™.

The client devices 110 are configured to communicate via a network 120,which may include any combination of local area and/or wide areanetworks, using both wired and/or wireless communication systems. In oneembodiment, a network 120 uses standard communications technologiesand/or protocols. For example, a network 120 includes communicationlinks using technologies such as Ethernet, 802.11, worldwideinteroperability for microwave access (WiMAX), 3G, 4G, code divisionmultiple access (CDMA), digital subscriber line (DSL), etc. Examples ofnetworking protocols used for communicating via the network 120 includemultiprotocol label switching (MPLS), transmission controlprotocol/Internet protocol (TCP/IP), hypertext transport protocol(HTTP), simple mail transfer protocol (SMTP), and file transfer protocol(FTP). Data exchanged over a network 120 may be represented using anysuitable format, such as hypertext markup language (HTML) or extensiblemarkup language (XML). In some embodiments, all or some of thecommunication links of a network 120 may be encrypted using any suitabletechnique or techniques.

The scanning engine 115 is a third-party system that detects andextracts text from images. The scanning engine 115 may extract text frombounding boxes within the images as single strings, words, blocks,and/or paragraphs. The scanning engine 115 also provides the location ofeach bounding box within the image. For example, the location of eachbounding box may be relative to an origin of the image (e.g., a top leftcorner, image center, etc.), relative to the margins of the image, orrelative to other bounding boxes within the image. In some embodiments,a set of coordinates describing a perimeter of each bounding box isprovided to the user 105. In other embodiments, coordinates describe oneor more corners of the bounding box, a side of the bounding box, or anyappropriate bounding box locus. The scanning engine 115 may also extracttext from form documents as a grid. For example, the scanning engine 115may segment a form document into an n-dimensional grid and identify thelocation of each string, word, etc., relative to its location within thegrid. It should be noted that in some embodiments, the scanning engine115 or software with similar functionality to the scanning engine may beintegrated into the image processing engine 125.

The image processing engine 125 extracts data from form documents and/orimages of form documents. Form documents include information relevant tothe user 105, and/or its employees, personnel, customers, andvolunteers, etc. Some or all data may be extracted by the imageprocessing engine 125 based on user preferences and engine settings.Examples of user preferences may include data extraction from certainfields within a form document, preferred separation characters, gridsize, etc. Examples of engine settings may include the use of separationcharacters for identifying separate text portions, scanning grid size,settings governing the upload of images, formats for presenting scannedtext and location information, and the like. Data extracted from animage of a form document may be stored in an electronic database, andmay also be used to generate electronic versions of the form documentsfor easy access and sharing among users. Form documents and/or images ofform documents may be provided via scan, fax, email, instant message,text/multimedia message, or other electronic conveyance by a user 105through a user interface or API of the image processing engine 125.

Form documents and images of form documents may vary in quality, evenbetween form documents of the same type. For example, images of formdocuments may be shifted or rotated relative to the image duringscanning, images may be scanned with different levels of resolution,images may be uploaded with different margin sizes, images may beuploaded at different times and/or by different users, and the like.Variation may be a result of manual scanning or faxing, variations inequipment used to provide images of form documents to the imageprocessing engine 125, user error, and the like. To account for thisvariation, and to automatically extract data from the form documents,the image processing engine 125 performs a calibration on a templateform document, and modifies completed form documents of the same formdocument type until the coordinates or locations of the text in thecompleted form documents matches or is similar to those in the templateform document. A template form document may be an empty form document, amarked-up form document, etc.

Based on the calibration, the image processing engine 125 can extractdata from completed form documents. To do this, the image processingengine 125 exports a template form document as an image to the scanningengine 115. The scanning engine provides the extracted text from thetemplate form document and the corresponding locations of the extractedtext within the image. The image processing engine 125 identifies afeature within the template form document that will serve as a map totransform the coordinates in completed form documents of the same typeas the template form document. Data is then extracted from the completedform documents using the transformed coordinates of the completed formdocuments, for subsequent use, display, and/or storage.

Image Processing Engine

FIG. 2 is a block diagram of an architecture of the image processingengine 125, according to one embodiment. The image processing engine 125shown in FIG. 2 includes a user interface 205, a form store 210, afeature extraction engine 215, a rotation engine 220, and a zoom engine225. In other embodiments, the image processing engine 125 may includeadditional, fewer, or different components for various applications.Conventional components such as security functions, load balancers,failover servers, management and network operations consoles, and thelike are not shown so as to not obscure the details of the systemarchitecture.

The user interface 205 allows a user 105 to interact with the imageprocessing engine 125. Through the user interface 205, a user 105 canupload form document images, view and modify data extracted from formdocuments, select form document features, identify templates, and thelike.

A form document may include declarative information about an employeethat was provided by the employee, for instance at the time of hiringthe employee. Each form document includes multiple fields, each for theentry of information corresponding to one or more attributes of anemployee. Examples of form documents include new hire form documents,emergency contact form documents, medical form documents, tax documents,insurance documents, etc. Each field either includes or is adjacent toform text associated with the field. For example, a “name” field, a“date of birth” field, a “primary cardholder” field, etc. Each fieldalso includes a space for entered text submitted by the employee or user105. For example, a user 105 may enter “Ariel,” April 19, 2001,” and“Rikki” for the “name,” “date of birth,” and “primary cardholder”fields, respectively. It should be noted that form documents may alsoinclude fields that do not include or are adjacent to text, though forthe purposes of simplicity, such fields aren't discussed further. Inaddition, it should be noted that although various form documentsdescribed herein are associated with employees or employers, inpractice, the functionalities described herein can be performed on anytype of form document.

The form store 210 stores form documents and images of form documents,for instance from a user 105 of the image processing engine 125. Theform store 210 also stores extracted form text and entered text of eachform document, the location of extracted form and entered text, metadataassociated with the form documents and form document images, and userpreferences. Metadata may include the data and time a form document wasprocessed, document author, form document type, form documentidentification number, modifications performed, identity of the user105, employee identity or employer identity, and the like. Userpreferences may include which form document and feature should be usedduring calibration, which data is extracted from each form document, howdata is stored, a threshold level of similarity between templates andcompleted form documents, and the like.

The feature extraction engine 215 identifies two or more orderedfeatures in template form documents that can be used as candidatefeatures for calibration. Candidate features include unique,identifiable strings of form text in the form document that can be usedto set the calibration parameters (e.g., a calibration distance and acalibration angle). For example, candidate features may include words,symbols, logos, and the like. The features ideally do not appear morethan once in the template form document; however, portions of thefeatures may appear more than once in the template form document. Thefeature extraction engine 215 identifies features for calibration andcomputes a calibration distance and a calibration angle. In otherembodiments, the user 105 may identify candidate features on the userinterface 205 of the image processing engine 125. The calibrationdistance is the distance between the two or more features and will beused to determine the percent enlargement required for each completedform document. The calibration angle is the angle of a line intersectingthe two or more features and will be used to determine the degree ofrotation required for each completed form document.

The calibration distance and calibration angle are computed usingcoordinates of the feature within the image of the form documentretrieved from the scanning engine 115. To retrieve the coordinates ofthe feature, the feature extraction engine 215 exports an image of atemplate form document to the scanning engine 115 and receives the textand coordinates from some or all fields in the template form document.The feature extraction engine 215 identifies coordinates of the two ormore features and computes the calibration distance and calibrationangle. The calibration distance and calibration angle may be calculatedusing a single pair of coordinates for each feature, a set ofcoordinates describing the bounding box of each feature, a portion ofthe bounding box, etc. The feature extraction engine 215 provides thecomputed calibration distance and calibration angle to the rotationengine 220, zoom engine 225, and/or form store 210. In some embodiments,the feature extraction engine 215 may use the extracted text andcoordinates of the template form document to generate an electronicversion of the template form document. Electronic versions of the formdocuments can then be populated with information extracted fromcompleted form documents and stored in the form store 210. From the formstore 210, a user 105 may access, modify, and share data from the formdocuments using the user interface 205.

The feature extraction engine 215 also retrieves data from completedform documents. The feature extraction engine 215 exports completed formdocuments to the scanning engine 115, and imports the extracted formtext, entered text, and location information describing locations offeatures and text within the completed form document. In someembodiments, the feature extraction engine 215 calculates the distancebetween the two or more features (“form feature distance”) and the angleof the line intersecting the two or more features (“form featureangle”). In these embodiments, the feature extraction engine 215provides the calculated form feature angle to the rotation engine 220and the calculated form feature distance to the zoom engine 225. Inother embodiments, the feature extraction engine 215 provides thecoordinates of the features to the rotation engine 220 and/or zoomengine 225. In these embodiments, the rotation engine 220 uses thecoordinates of the features to compute form feature angle, and the zoomengine 225 uses the coordinates to compute the form feature distance. Asdescribed below, the rotation engine 220 and zoom engine 225 perform arotation operation and a zoom operation on the image of the formdocument using the form feature angle and form feature distance, and thecalibration angle and calibration distance.

The rotation engine 220 and the zoom engine 225 use the form featureangle, the form feature distance, the calibration angle, and calibrationdistance to zoom and rotate the image of the form document in one of atleast two ways. In a first embodiment, the rotation engine 220 rotatesthe image of the form document based on a difference between the formfeature angle and the calibration angle, and the zoom engine 225 zoomsthe rotated image of the form document based on a difference between theform feature distance and the calibration distance. The zoomed androtated image of the form document is then sent to the scanning engine115 for re-scanning, and the scanning engine provides updated locationsfor features and text of the zoomed and rotated image. In a secondembodiment, the rotation engine 220 modifies the coordinates of eachfeature and/or text within the image of the form document based on oneor more of 1) a difference between the form feature angle and thecalibration angle, 2) based on a difference between the form featuredistance and the calibration distance, 3) based on a location of aparticular feature or text relative to a reference point of the image ofthe form document, and 4) based on a distance between the location ofthe particular feature or text relative to the reference point of theimage of the form document. In such an embodiment, the location orcoordinates of features and/or text of the image of the form documentcan be transformed such that each field of the image of the formdocument can correspond to an associated field of the template formdocument, and the text of such fields can be extracted without requiringthe image of the form document to be rescanned by the scanning engine115.

The rotation engine 220 computes the degree of rotation required totransform completed form documents to match the calibrated template formdocument. To do this, the rotation engine 220 compares the calibrationangle to the form feature angle. If the difference between thecalibration angle and form feature angle is greater than a thresholdangle, the rotation engine 220 rotates the image of the completed formdocument and/or coordinates of the completed form document until thedifference is within a threshold angle. The rotation may be based on theposition of each word in the form document, the position of each wordrelative to the features, the position of each word relative to theimage, etc. Likewise, the rotation engine 220 may rotate the image untila top of the form is aligned with a top of the image, or may rotate theimage by an amount equal to the form feature angle (e.g., in embodimentswhere the form feature angle is measured relative to a properly alignedform document). In some embodiments, the rotated image and/orcoordinates of the image are provided to the zoom engine 225. In otherembodiments, images of form documents are first processed by the zoomengine 225 and then by the rotation engine 220.

The zoom engine 225 computes a percentage that a completed form documentmust be enlarged or shrunk to match the calibrated template formdocument. The zoom engine 225 compares the form feature distance of eachcompleted form document to the calibration distance. If the form featuredistance is smaller than the calibration distance is greater than athreshold distance, zoom engine 225 enlarges the image and/orcoordinates of the completed form document until the difference is lessthan the threshold distance. If the form feature distance is greaterthan the calibration distance by more than a threshold distance, zoomengine 225 shrinks the image and/or coordinates of the completed formdocument until the difference is less than the threshold distance. Insome embodiments, different or additional operations may be performed onthe image of the form document to align it with the image of thetemplate form document. For example, a shift operation may be performedon the image and/or coordinates of the completed form document when thefeatures in the image of the completed form document are shiftedrelative to the features in the image of the template form document bymore than a threshold distance.

The enlarged and rotated image and/or coordinates are provided to thefeature extraction engine 215. The feature extraction engine 215 usesthe coordinates to identify the entered text for each field in thecompleted form documents. The entered text may be stored in the formstore 210 and provided to the user for display on an element of the userinterface 205. The entered text may also be used to generate electronicversions of the completed form documents. The electronic form documentsmay be stored in the form store 210 or in a third-party database fromwhere they can be accessed by a user 105. In some embodiments, theenlarged and rotated image is provided to the scanning engine 115 toobtain additional coordinates describing features, fields, and text ofthe enlarged and rotated image.

FIGS. 3A-3D illustrate the process of transforming an image of a formdocument, according to one embodiment. FIG. 3A illustrates an image 305of a template form document 310. In the example shown, the template formdocument 310 is fully within the margins of the image 305, has the sameorientation as the image 305, and fills the image 305 without cuttingoff any portion of the template form document 310. In other examples,the template form document 310 may be shifted within the image 305and/or rotated within the image 305. The template form document includesmultiple fields, including the “name” field 315, “M.I.” field 317, and“SSN” field 320. Form text, e.g., “name,” “M.I,” and “SSN” may belocated within the field, as shown in the “name” field 315 and in the“M.I.” field 317, or it may be located adjacent to the field, as shownin the “SSN” field 320. The template form document 310 does not includeentered text (i.e., text entered by a user, employee, volunteer, etc.);however, template form documents may contain entered text.

The scanning engine 215 determines the text and coordinates of textwithin the image 305, and the feature extraction engine 215 identifieswhich coordinates correspond to the features. In template form documentswhere a portion of the features appears more than once in the form, thefeature extraction engine 215 may use Hough transforms or other suitablefeature identification algorithms to identify the location of thefeatures. For example, features may include “DOB,” “address,” and“phone” and other instances of “address” and “phone” appear within theform. The feature extraction engine 215 identifies which instances of“address” and “phone” correspond to the features. The features in thetemplate form document 310 contains the “name” field 315 and the “M.I.”field 317. In other examples, features may include different oradditional words, symbols and/or logos. Additionally, features may beadjacent vertically and/or horizontally.

σ₁ 325, represented as a dashed line, is the calibration angle of theimage 305 of the template form document 310. In the illustration shown,σ₁ 325 is the angle of a line intersecting the “name” field 315 and the“M.I.” field 317. σ₁ 325 may be calculated using some or all of thecoordinates that describe the bounding box around the features, e.g.,the boxes around “name” field 315 and the “M.I.” field 317. σ₁ 325 mayalso be calculated using any coordinates that represent the location ofthe features within the image 305.

D₁ 330 is the calibration distance of the image 305 of the template formdocument 310. In the illustration shown in FIG. 3A, D₁ 330 is thedistance between the “name” field 315 and the “M.I.” field 317. D₁ 330is calculated from the bottom left corner of “name” field 315 and thebottom left corner of the “M.I.” field 317. D₁ 330 D₁ 330 may also becalculated using any coordinates that represent the location of thefeatures within the image 305. For example, D₁ 330 may be calculatedfrom the mid-point of each field, the inner corners of the fields, theouter corners of the fields, and the like. In other examples, D₁ 330 maybe calculated as the width of the margins between the edge of the image305 and the features within the template form document 310. For example,D₁ 330 may be the distance between a left margin of the image 305 andthe “name” field 315, and a second distance may be calculated between aright margin of the image and the “M.I.” field 317.

FIG. 3B illustrates an image 335 of a completed form document 340. Thecompleted form document 340 is a version of the template form document310 in which entered text has been inputted into a portion of thefields. For example, the “name” field 345 of the completed form document340 includes the name “Nate” and the “M.I.” field 347 of the completedform document 340 includes the letter “B.” As shown in the FIG. 3B, thesize of the image 335 of the completed form document 340 is the samesize of the image 305 of the template form document 310. In otherembodiments, the image of template form document does not have to be thesame size as the image of the completed form document. Similarly, otherimage characteristics do not need to be similar across the images oftemplate and completed form documents, e.g., the resolution,orientation, aspect ratio, color, etc.

The size and orientation of the completed form document 340 within itsrespective image, e.g., image 335, are not the same as those of thetemplate form document 310 within its respective image, e.g., image 305.In particular, the completed form document 340 is rotatedcounterclockwise within the frame of the image 335 and is smallerrelative to the template form document 310. Variations in size,orientation, and image quality may be a result of the user how scannedthe form documents, the equipment used to scan the form documents, thethird-party software, etc.

σ₂ 350, represented as a dashed line, is the form feature angle of thecompleted form document 340. The difference in the calibration angle, σ₁325, and the form feature angle, σ₂ 350, is computed by the rotationengine 220. In some embodiments, if the difference is greater than thepredetermined threshold, the rotation engine 220 rotates the image 335of the completed form document 340 until σ₂ 350 is within a thresholdangle of σ₁ 325.

D₂ 355 is the form feature distance of the completed form document 340.The difference in the calibration distance, D₁ 330, and the form featuredistance, D₂ 355, is computed by the zoom engine 225. In someembodiments, if the difference is greater than the predeterminedthreshold, the zoom engine 225 enlarges the image 335 of the completedform document 340 until D₂ 355 is within a threshold distance of D₁ 330.

FIG. 3C illustrates an enlarged image 337 of the completed form document340. Based on the difference between the calibration distance, D₁ 330,and the form feature distance, D₂ 355, the zoom engine 225 transformsthe image of the form feature document and/or the coordinates of allform text and entered text within the image 337 of the completed formdocument 340 until the form feature distance, D₂ 355, is within athreshold distance of the calibration distance, D₁ 330. As shown, theimage of the completed form document 340, e.g., image 337, is enlargeduntil the distance between the “name” field 345 and “M.I.” field 347 isD₁ 330.

FIG. 3D illustrates an enlarged and rotated image 339 of the completedform document 340. The rotation engine 220 rotates the image of thecompleted form document 340 or transforms the coordinates of all formtext and entered text within the image 339 of the completed formdocument 340 until the form feature angle, σ₂ 350, is within a thresholdangle of calibration angle, σ₁ 325. As shown, the image of the completedform document 340, e.g., image 339, is rotated until the angle of a lineintersecting the “name” field 345 and “M.I.” field 347 is σ₁ 325. Insome embodiments, different or additional operations may be performed onthe image of the completed form document. For example, a shift operationmay be performed on the image of the completed form document to alignthe image of the form document with the image of the template formdocument.

In some embodiments, the transformed image may be sent to the scanningengine 115 in order for the feature extraction engine 215 to retrievethe form text, entered text, and corresponding coordinates of thecompleted form documents. The feature extraction engine 21 may thenextract data from the completed form documents. In other embodiments,the transformed coordinates may be sent to the feature extraction engine215 to directly extract data from the completed form documents.

In other embodiments, form text and entered text are extracted from formdocuments without transforming the image of the form document. In theseembodiments, the coordinates of the form text and location text of theimage of the form document are transformed using the calibration angle,calibration distance, form feature angle, and form feature distance. Forexample, if the difference between the calibration angle and formfeature angle is greater than a predetermined threshold, the rotationengine 220 recomputes the coordinates of the form text and entered textin the form document until the form feature angle is within a thresholdangle of the calibration angle. Further, if the difference in thecalibration distance and the form feature distance is greater than apredetermined threshold, the zoom engine 225 recomputes the coordinatesof the form text and entered text until the form feature angle is withina threshold distance of the calibration angle. In these embodiments,form text and entered text can be extracted from images of formdocuments without re-sending the images of form documents to thescanning engine 115.

FIG. 4 is a flow chart illustrating a method 400 of extracting data froman image of a form document, according to one embodiment. An image of aform document is accessed 405 by the image processing engine 125. Insome embodiments, images of form documents are provided by a user 105 ofthe image processing engine 125. In other embodiments, images of formdocuments are accessed 405 from a form store 210 or from a third-partyelectronic database. Form documents include fields, and each fieldincludes form text and entered text. For each of a first set of formtext and a second set of form text, a first distance between the firstset of form text and second set of form text is accessed 410. The firstdistance may be accessed 410 from the feature extraction engine 215, thescanning engine 115, the form store 210, etc. In some embodiments, thefirst distance may be a calibration distance computed by the featureextraction engine 215 using a template form document, the image of theform document, a different image of a different form document, etc. Foreach of the first set of form text and the second set of form text, afirst angle of a line interesting the first set of form text and thesecond set of form text is accessed 415. The first angle may be accessed410 from the feature extraction engine 215, the scanning engine 115, theform store 210, etc. In some embodiments, a first angle may be acalibration angle computed by the feature extraction engine 215 using atemplate form, the image of the form document, a different image of adifferent form document, and the like.

For each of the first set of form text and second set of form text, acorresponding set of image coordinates are determined 420. The imagecoordinates represent the location of the form text relative to theimage of the form document. In some embodiments, the image coordinatesrepresent a perimeter of a bounding box around form text, an edge of abounding box around the form text, a corner of a bounding box around theform text, etc. A second distance between the set of image coordinatescorresponding to the first set of form text and the set of imagecoordinates corresponding to the second set of form text is computed425. Similarly, a second angle of a line intersecting the set of imagecoordinates between the set of image coordinates corresponding to thefirst set of form text and the set of image coordinates corresponding tothe second set of form text is computed 430. A zoom operation on theimage of the form document is performed 435 based on a comparisonbetween the first distance and second distance to produce a zoomed formdocument image. A rotation operation on the image of the form documentis performed 440 based on a comparison between the first angle andsecond angle to produce a modified form document image. In someembodiments, a shift operation may also be performed on the image of theform document to align the image of the completed document to an imageof a template form document. Text is then extracted from the modifiedform document image, and each portion of extracted text is associatedwith a field of the form document that aligns or corresponds to thelocation of the portion of extracted text within the modified formdocument. The extracted text can then be used, displayed, modified, orstored for subsequent use and access.

CONCLUSION

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

Some portions of this description describe the embodiments in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are commonly used bythose skilled in the data processing arts to convey the substance oftheir work effectively to others skilled in the art. These operations,while described functionally, computationally, or logically, areunderstood to be implemented by computer programs or equivalentelectrical circuits, microcode, or the like. Furthermore, it has alsoproven convenient at times, to refer to these arrangements of operationsas modules, without loss of generality. The described operations andtheir associated modules may be embodied in software, firmware,hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, and/or it may include a general-purpose computingdevice selectively activated or reconfigured by a computer programstored in the computer. Such a computer program may be stored in anon-transitory, tangible computer readable storage medium, or any typeof media suitable for storing electronic instructions, which may becoupled to a computer system bus. Furthermore, any computing systemsreferred to in the specification may include a single processor or maybe architectures employing multiple processor designs for increasedcomputing capability.

Embodiments may also relate to a product that is produced by a computingprocess described herein. Such a product may include informationresulting from a computing process, where the information is stored on anon-transitory, tangible computer readable storage medium and mayinclude any embodiment of a computer program product or other datacombination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the patent rights. It istherefore intended that the scope of the patent rights be limited not bythis detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thepatent rights, which is set forth in the following claims.

What is claimed is:
 1. A method comprising: accessing an image of a formdocument comprising a plurality of fields, wherein each field includesor is adjacent to a set of form text associated with the field, andwherein each field includes entered text; computing a distance between afirst set of form text and a second set of form text within the image ofthe form document; computing an angle of a line intersecting the firstset of form text and the second set of form text; and modifying theimage of the form document by: performing a zoom operation on the imageof the form document based on a comparison of the computed distance andan expected distance between the first set of form text and the secondset of form text to produce a zoomed form document image; and performinga rotation operation on the zoomed form document image based on acomparison of the computed angle and an expected angle between the firstset of form text and the second set of form text to produce a modifiedform document image.
 2. The method of claim 1, wherein the first set ofform text is associated with a first bounding box around the first setof form text and the second set of form text is associated with a secondbounding box around the second set of form text within the image.
 3. Themethod of claim 2, wherein the computed distance comprises a distancebetween a corner of the first bounding box and a corresponding corner ofa second bounding box.
 4. The method of claim 1, wherein performing thezoom operation comprises: determining a difference between the computeddistance and the expected distance; and if the difference is greaterthan a threshold difference, modifying dimensions of the image by afactor selected based on the determined difference.
 5. The method ofclaim 1, wherein performing the rotation operation comprises:determining a difference between the computed angle and the expectedangle; and if the difference is greater than a threshold difference,modifying an orientation of the image by a factor selected based on thedetermined difference.
 6. The method of claim 1, further comprising:retrieving entered text from a field of the modified form documentimage; identifying a field of the form document corresponding to thefield of the modified form document image; and modifying a databasefield corresponding to the identified field of the form document toinclude the retrieved entered text.
 7. The method of claim 1, furthercomprising: automatically scanning the entered text of each field fromthe modified form document image.
 8. The method of claim 1, whereincomputing a second distance between the image coordinates comprisesapplying a Hough transform to the image coordinates.
 9. A non-transitorycomputer-readable storage medium containing computer program code that,when executed by a processor, causes the processor to perform stepscomprising: accessing an image of a form document comprising a pluralityof fields, wherein each field includes or is adjacent to a set of formtext associated with the field, and wherein each field includes enteredtext; computing a distance between a first set of form text and a secondset of form text within the image of the form document; computing anangle of a line intersecting the first set of form text and the secondset of form text; and modifying the image of the form document by:performing a zoom operation on the image of the form document based on acomparison of the computed distance and an expected distance between thefirst set of form text and the second set of form text to produce azoomed form document image; and performing a rotation operation on thezoomed form document image based on a comparison of the computed angleand an expected angle between the first set of form text and the secondset of form text to produce a modified form document image.
 10. Thenon-transitory computer-readable storage medium of claim 9, wherein thefirst set of form text is associated with a first bounding box aroundthe first set of form text and the second set of form text is associatedwith a second bounding box around the second set of form text within theimage.
 11. The non-transitory computer-readable storage medium of claim10, wherein the computed distance comprises a distance between a cornerof the first bounding box and a corresponding corner of a secondbounding box.
 12. The non-transitory computer-readable storage medium ofclaim 9, wherein performing the zoom operation comprises: determining adifference between the computed distance and the expected distance; andif the difference is greater than a threshold difference, modifyingdimensions of the image by a factor selected based on the determineddifference.
 13. The non-transitory computer-readable storage medium ofclaim 9, wherein performing the rotation operation comprises:determining a difference between the computed angle and the expectedangle; and if the difference is greater than a threshold difference,modifying an orientation of the image by a factor selected based on thedetermined difference.
 14. The non-transitory computer-readable storagemedium of claim 9, wherein the program code, when executed by theprocessor, causes the processor to perform further steps comprising:retrieving entered text from a field of the modified form documentimage; identifying a field of the form document corresponding to thefield of the modified form document image; and modifying a databasefield corresponding to the identified field of the form document toinclude the retrieved entered text.
 15. A system comprising: a hardwareprocessor; and a non-transitory computer-readable medium containinginstructions that, when executed by the hardware processor, cause thehardware processor to: accessing an image of a form document comprisinga plurality of fields, wherein each field includes or is adjacent to aset of form text associated with the field, and wherein each fieldincludes entered text; computing a distance between a first set of formtext and a second set of form text within the image of the formdocument; computing an angle of a line intersecting the first set ofform text and the second set of form text; and modifying the image ofthe form document by: performing a zoom operation on the image of theform document based on a comparison of the computed distance and anexpected distance between the first set of form text and the second setof form text to produce a zoomed form document image; and performing arotation operation on the zoomed form document image based on acomparison of the computed angle and an expected angle between the firstset of form text and the second set of form text to produce a modifiedform document image.
 16. The system of claim 15, wherein the first setof form text is associated with a first bounding box around the firstset of form text and the second set of form text is associated with asecond bounding box around the second set of form text within the image.17. The system of claim 16, wherein the computed distance comprises adistance between a corner of the first bounding box and a correspondingcorner of a second bounding box.
 18. The system of claim 15, whereinperforming the zoom operation comprises: determining a differencebetween the computed distance and the expected distance; and if thedifference is greater than a threshold difference, modifying dimensionsof the image by a factor selected based on the determined difference.19. The system of claim 15, wherein performing the rotation operationcomprises: determining a difference between the computed angle and theexpected angle; and if the difference is greater than a thresholddifference, modifying an orientation of the image by a factor selectedbased on the determined difference.
 20. The system of claim 15, furthercontaining instructions that cause the hardware processor to: retrieveentered text from a field of the modified form document image; identifya field of the form document corresponding to the field of the modifiedform document image; and modify a database field corresponding to theidentified field of the form document to include the retrieved enteredtext.